HFSP: The Hadoop Fair Sojourn Protocol

This work presents the HFSP scheduler, which implements a size-based scheduling discipline for Hadoop. While the benefits of size-based scheduling disciplines are well recognized in a variety of contexts (computer networks, operating systems, etc...), their practical implementation for a system such as Hadoop raises a number of important challenges. In HFSP we address issues related to job size estimation, resource management and study the effects of a variety of preemption strategies. Although the architecture underlying HFSP is suitable for any size-based scheduling discipline, in this work we revisit and extend the Fair Sojourn Protocol, which solves many problems related to job starvation that affect FIFO, Processor Sharing and a range of size-based disciplines. Our experiments, in which we compare HFSP to standard Hadoop schedulers, pinpoint at a significant decrease in average job sojourn times – a metric that accounts for the total time a job spends in the system, including waiting and serving times – for realistic workloads that we generate according to production workload traces available in the literature.